The present invention relates generally to image processing, and more particularly relates to a system and methods that allow the user of an image processing system to implement scene-by-scene color manipulation in the primary color domain to color correction regions of a video image isolated in the hue domain using hue, saturation, and luminance qualification.
In a video signal color correction system, various types of image processing are often employed to create, enhance, compress, filter, or otherwise modify characteristics of the video image. In certain types of video image processing systems, especially post-production color correction systems for motion picture film and/or video tape, color corrections are typically made on a scene-by-scene basis. A xe2x80x9cscenexe2x80x9d is a sequential collection of images, often shot from the same camera, having the same viewpoint, composed in a certain way, etc. An operator using a typical post-production color correction system observes a target frame of the scene on a video monitor, adjusts the color and other parameters of the frame until it is aesthetically satisfactory, and stores the color correction parameters in system memory. The color correction system preferably automates the application of the stored color correction parameters to the other frames of the scene.
For example, the system operator or xe2x80x9ccoloristxe2x80x9d typically selects a scene to manipulate, and then selects a particular frame from the scene for manual manipulation. The colorist views the target frame as a still image on the system monitor and applies color corrections via a control panel to adjust the color parameters for a particular region of the target frame. The color correction system typically allows the colorist to isolate a particular region of the frame and to alter the intensity of the primary color components, red, green and blue, which in various combinations produce all of the colors that the system can produce. The colorist may than apply color corrections to another region of the frame, and so forth, until all of the desired regions have been color corrected. The correction settings are then stored in system memory.
After the colorist is satisfied with the adjustments made to the target frame, the color correction system, which is typically computer-controlled, automatically applies the stored color corrections to each frame in the scene on a frame-by-frame basis. The color corrected frames of the scene are then recorded on film or videotape. The steps are repeated for other scenes in the film or video tape, often with different correction settings stored for different scenes. This process may be repeated as needed to create a color corrected master film or video tape that reflects multiple color adjustments to multiple frames in multiple scenes. In the general case, multiple color adjustments may be applied to all of the frames of a motion picture or other video program.
Devices are known in the art for isolating a region of a still image for applying one set of color corrections, with other regions of the image receiving a different set of color corrections. These devices allow a color correction system to isolate a particular region of a frame to receive special image processing. For example, when color correcting a soft drink commercial it may be desirable to isolate the beverage container from the remainder of the image. The color of the beverage container may then be enhanced to make the can stand out from the rest of the image. But applying color correction to each frame of a film or video tape individually is extremely tedious and time consuming. Automating the process of applying the color correction parameters defined for one frame of a scene to the other frames is, therefore, highly desirable.
Automatically isolating a color correction region presents a difficult technical problem because the region of interest may change in size, shape, location, and/or geometry over the frames in a scene. That is, an object in a video scene typically moves over the frames of the scene. For example, consider a scene in which a bottle is lifted, tilted, moved toward a glass, turned toward the viewer, and then further tilted to pour the contents of the bottle into a glass. In this scene, the geometry of the bottle changes from a side view of the bottle (an irregular shape) to a top view of the bottle (an essentially round shape). The bottle also changes in location and size over the several frames of the scene.
The state of the art in automated color correction technology is somewhat lacking for a number of reasons. Video images are typically defined on a pixel-by-pixel basis by intensity levels of the primary colors, red, green and blue (R,G,B). In a typical color correction system, the primary colors are mixed in varying intensities to produce all of the colors that the system can produce. Color data in the R,G,B format, often referred to as the xe2x80x9cprimary color domain,xe2x80x9d may be linearly transformed into secondary color components, hue, saturation and luminance, which is sometimes referred to as the xe2x80x9chue domain.xe2x80x9d Certain prior art color correction systems apply color corrections in the primary color domain, whereas others apply color corrections in hue domain.
Operating in the primary color domain has certain drawbacks associated with isolating color correction regions. In a color video image, the intensity of the red, green, and blue components for a particular object depicted in a scene can vary from frame to frame in response to the brightness, or shadowing, of the item. In addition, many different colors may have the same intensity of one of the particular color components. For this reason, it is difficult to isolate a particular object in a scene by focusing on the intensity of the red, green, and blue constituents of the pixels of the object. In other words, isolating a particular object, as that object moves from frame to frame within a scene, is difficult when analyzing the video data in the primary color domain.
Color correction systems that operate in the hue domain have been developed to overcome the region isolation drawbacks associated with color correction systems that operate in the primary color domain. For example, color correction systems have been developed that allow an operator to select and manipulate a color correction region in the hue domain. These systems typically include equipment for generating and positioning a cursor on a video monitor to allow selection of a color correction region defined by the hue at the selected position. Circuitry responsive to the cursor location selects one of a plurality of color correction circuits to become operative for directing secondary color correction parameters (i.e., adjustments to the hue, saturation, and luminance) only to regions in the video image corresponding to the hue selected by the cursor. This type of system therefore allows application of secondary color correction parameters to all regions of the image bearing the hue that was selected with the cursor.
Color correction systems that operate in the hue domain, however, also exhibit certain drawbacks. First, they incur a relatively high processing overhead because the entire video image is typically transformed from the primary color domain to the hue domain. Color corrections are then applied in the hue domain, and the image is then transformed back from the hue domain to the primary color domain. Thus, the system performs two linear transformations on the entire video image even though no color corrections may be applied to large portions of the image. As a result, even those portions of the video image that are not color corrected must be transformed from the primary color domain to the hue domain, and then back to the primary color domain.
Second, transforming color data from the primary color domain to the hue domain, and then back to the primary color domain, can impart xe2x80x9ccolor artifactsxe2x80x9d into the video image. That is, applying a linear mathematical operation to digital data, followed by the inverse mathematical operation, can alter the data for a small percentage of pixels. This phenomenon occurs due to the truncation of irrational numbers in floating-point arithmetic. Stated differently, due to the limited precision of linear mathematical operations applied to digital data, applying an arithmetic operation to digital data, followed by the inverse arithmetic operation, can alter the values for some of the data points. Although the vast majority of data points are not altered by the arithmetic operations, a small percentage of data points are typically altered. These altered data points are known as xe2x80x9ccolor artifacts.xe2x80x9d
To avoid the combined drawbacks associated with region isolation in the primary color domain and color artifacts created through operation in the hue domain, color correction systems have been developed that apply color corrections in the primary color domain using geometric constraints to isolate color correction regions. For example, certain color correction systems include a video parameter control system operative for selecting a spatial region or window in a video image for color correction. A track ball allows selection of a spatial region by dragging a cursor to draw a window around an object of interest in the picture. The operator then adjusts controls that affect only the selected window or region. A first set of correction signals may be stored for the selected region, and a separate second set of correction signals may be stored for areas of the picture outside the selected region, thereby allowing multiple sets of corrections for a given frame. Although these systems provide some control over the movement of the window over a plurality of frames in a scene, the region is essentially static and the geometry is invariably that of a box.
Other prior art image processing computer software known as the xe2x80x9cSimple Windowsxe2x80x9d and xe2x80x9cPower Windowsxe2x80x9d features are extensions of the windows features of previous system. These features are provided in the RENAISSANCE 8:8:8(trademark) digital color enhancement system, manufactured by the assignee of the present invention. In the Simple Windows feature, a window is a predetermined regularly shaped area or region of the video image that can be varied in size. The colors within the window are independently adjustable from the colors of the rest of the image. Primary or secondary color enhancements can occur both inside and outside a window, and each adjustment is independent of the other. The Simple Windows feature entails use of a simple geometric formxe2x80x94a square or rectanglexe2x80x94for a window shape. A Simple window is always rectangular and is defined by four points. Lines defining the window are always straight vertical or horizontal. Furthermore, there are no soft edges, that is, there is a sharp delineation between the inside and outside of the window, which sometimes produces undesirable image effects at the boundaries.
The more recent xe2x80x9cPower Windowsxe2x80x9d feature provides more choices of the shape for the window, for example, circular, rectangular, half screen, split in the middle, etc. Using this system, the operator can select windows including multiple squares, multiple diamonds (essentially rotated squares), horizontal and vertical bars, circles, ellipses (a warped circle), and so forth. The sizes of these windows can be varied so long as the shape remains regular. Furthermore, a xe2x80x9csoft edgesxe2x80x9d feature is available to provide a gradual transition in color correction from the inside to the outside of a selected window. Power Windows, however, are still confined to regular shapes, with predefined geometry.
The commonly owned patent application Ser. No. 08/912,662, entitled xe2x80x9cUser Defined Windows For Selecting Image Processing Regions,xe2x80x9d filed Aug. 18, 1997 by Xueming Henry Gu, et al., describes further improvements in the RENAISSANCE 8:8:8(trademark) digital color enhancement system. This system allows the user to select color correction regions corresponding to objects in a target frame using a geometric masking technique. For example, the user may mask the same object in the first and last frames of a scene. The system then automatically xe2x80x9cmorphsxe2x80x9d or changes the shape of a selected object over the intervening frames of a scene. Although the system can automatically morph an object over a number of frames, the edges of the automatically-morphed object can diverge from the edges of the actual object in the scene. The system therefore includes a utility that allows the user to manually correct automatically-morphed objects on a frame-by-frame basis. While this utility is quite effective for correcting unacceptable divergence, manually imparting the required adjustments on a frame-by-frame basis can be tedious and time consuming.
In sum, there is a continuing need for a color correction system that allows the isolation of color correction regions that move and change size and geometry over several frames in a scene. There is a further need for a color correction system that isolates regions and implements color corrections without imparting color artifacts into the video image. And there is a need for further for automating the process of color correction region isolation to minimize or obviate the need for manual frame-by-frame adjustment of the processed video image.
The present invention overcomes the disadvantages described above in a color correction system that implements primary color manipulation using secondary color correction region isolation. As such, the system performs color correction in the primary color domain, which means that color data that is not manipulated passes through the system unaltered. This aspect of the invention prevents color artifacts from appearing in the processed video image and lowers the data processing overhead because no operations, other than timed buffering to maintain synchronism, are performed for unprocessed or pass-through pixels.
Although color correction is performed in the primary color domain, the system isolates color correction regions through operations performed in the hue domain. Specifically, hue qualification, saturation qualification, luminance qualification, and an optional alpha filter are combined to define an alpha qualification function that isolates a region or xe2x80x9chue sectorxe2x80x9d for color correction. The alpha qualification function may have a shape in the hue domain that tapers or softens the applied color correction towards the edges of the corrected hue sector. The system includes a user interface that allows the colorist to select the shape of the hue, saturation, and luminance alpha qualification functions, as well as the shape of the optional alpha filter, in the hue domain. This aspect of the invention gives the colorist fine control over the color isolation and correction process, including fine control over the sharpness of the color correction at the edges of color corrected hue sectors.
Color correction region isolation in the hue domain may be combined with geometric constrains. Geometric constraints allow the colorist to limit the automatic application of color correction to a defined portion of the frame area, over the several frames of a scene. As a mechanism for selectively applying color correction to the frames of a scene, geometric constraints may be used to increase the system""s ability to isolate discrete objects for color correction. For example, a colorist may use a geometric constraint to exclude from color correction discrete objects that have a similar color to a target object.
Generally described, the invention is a computer-implemented process for selectively applying image processing to an image. Color correction equipment receives an input signal from an image source in a primary color domain. The color correction equipment also receives a command selecting a sample of the input signal, and a command identifying a color correction parameter in the primary color domain associated with the sample. The color correction equipment determines a hue-domain parameter associated with the sample and a sector in the hue-domain about the hue-domain parameter associated with the sample. The color correction equipment then defines a qualification curve corresponding to the sector and mixes the input signal and the color correction parameter within the sector in accordance with the qualification curve. The hue-domain parameter may be selected from the group including hue, saturation, and luminance.
The color correction equipment may also determine a second hue-domain parameter associated with the sample and define a second qualification curve about the second hue-domain parameter and corresponding to the sector. The first and second qualification curves are then combined to obtain a combined qualification curve. In this case, the color correction mixes the input signal and the color correction parameter within the sector in accordance with the combined qualification curve. The first and second hue-domain parameters may be selected from the group including hue, saturation, and luminance.
In addition, the color correction equipment may determine a third hue-domain parameter associated with the sample and define a third qualification curve about the third hue-domain parameter and corresponding to the sector. The first, second, and third qualification curves are then combined to obtain a total qualification curve. In this case, the color correction equipment mixes the input signal and the color correction parameter within the sector in accordance with the total qualification curve. The first, second, and third hue-domain parameters may include the hue, saturation, and luminance.
The color correction equipment may also apply a filter to the total qualification curve to obtain an alpha qualification curve. If this case, the color correction equipment mixes the input signal and the color correction parameter within the sector in accordance with the alpha qualification curve. Moreover, the color correction equipment may receive a command defining a geometric constraint corresponding to the input signal. In this case, the color correction equipment limits the mixing the input signal and the color correction parameter in accordance with the geometric constraints.
The invention also provides a computer-implemented process for selectively applying image processing to a plurality of frames defining a scene. Color correction equipment receives an input signal defining the scene from the image source in a primary color domain. The color correction equipment also receives a command selecting a target frame of the scene and another command selecting a sample of the target frame. The color correction equipment also receives a command identifying a color correction parameter in the primary color domain associated with the sample. The color correction equipment then determines a hue-domain parameter associated with the sample and defines a sector in the hue-domain about the hue-domain parameter associated with the sample. The color correction equipment also defines a qualification curve corresponding to the sector. Then, for each frame of the scene, the color correction equipment mixes the input signal and the color correction parameter within the sector in accordance with the qualification curve.
These and other features and advantages of the invention may be more clearly understood and appreciated from a review of the following detailed description of the disclosed embodiment, and by reference to the appended drawings and claims.